The objective optics used in digital cameras are typically designed so as to minimize the optical point spread function (PSF) and maximize the modulation transfer function (MTF), subject to the limitations of size, cost, aperture size, and other factors imposed by the camera manufacturer. The PSF of the resulting optical system may still vary from the ideal due to focal variations and aberrations. A number of methods are known in the art for compensating for such PSF deviations by digital image processing.
For example, U.S. Pat. No. 6,154,574, whose disclosure is incorporated herein by reference, describes a method for digitally focusing an out-of-focus image in an image processing system. A mean step response is obtained by dividing a defocused image into sub-images, and calculating step responses with respect to the edge direction in each sub-image. The mean step response is used in calculating PSF coefficients, which are applied in turn to determine an image restoration transfer function. An in-focus image is obtained by multiplying this function by the out-of-focus image in the frequency domain.
PCT International Publication WO 2004/063989, whose disclosure is incorporated herein by reference, describes an electronic imaging camera, comprising an image sensing array and an image processor. The image processor applies a deblurring function—typically in the form of a deconvolution filter (DCF)—to the signal output by the array in order to generate an output image with reduced blur. This blur reduction makes it possible to design and use camera optics with a poor inherent PSF, while restoring the electronic image generated by the sensing array to give an acceptable output image.
Low-cost color video cameras typically use a single solid-state image sensor with a multi-colored mosaic filter overlay. A mosaic filter is a mask of miniature color filter elements in which a filter element is positioned in front of each detector element of the image sensor. For example, U.S. Pat. No. 4,697,208, whose disclosure is incorporated herein by reference, describes a color image pickup device that has a solid-state image sensing element and a complementary color type mosaic filter. Any sort of image sensor with a color mosaic filter, regardless of the choice and arrangement of the colors in the mosaic, is referred to hereinbelow as a “mosaic image sensor.”
The filter elements in the mosaic filter generally alternate between the primary RGB colors, or between the complementary colors cyan, magenta and yellow. One common type of color mosaic filter is called a “Bayer sensor” or “Bayer mosaic,” which has the following general form (in which letters represent colors—R denotes red, G denotes green and B denotes blue):
RGRGRGGBGBGBRGRGRGGBGBGBRGRGRGGBGBGBThe different color filters have respective passbands, which may overlap. The Bayer mosaic is described in U.S. Pat. No. 3,971,065, whose disclosure is incorporated herein by reference.
Processing the output of a mosaic image sensor typically involves reconstructing the full, color image by extracting three color signals (red, green and blue) from the sensor output. An image signal processor (ISP) processes the color signals in order to compute luminance (Y) and chrominance (C) values for each pixel of the output image. The ISP then outputs these values (or the corresponding R, G and B color values) in a standard video format.
The Standard Mobile Imaging Architecture (SMIA) is an open standard developed by Nokia and STMicroelectronics for use by companies making, buying or specifying miniature integrated camera modules for use in mobile applications. Information regarding this standard is available on the Internet at the SMIA-forum Web site. The main object of the standard is to make it possible to connect any SMIA-compliant sensor to any SMIA-compliant host system with matching capabilities and get a working system. The SMIA 1.0 Part 1: Functional Specification (2004) provides standardized electrical, control and image data interfaces for Bayer sensor modules, which output raw data to an image signal processor (ISP) or other processing host. Chapter 10 of the SMIA specification, which is incorporated herein by reference, describes a method for differential pulse code modulation (DPCM) that may be used to compress the image output of the sensor from ten to eight bits/pixel in order to reduce the required bandwidth in transmission between the sensor module and the host.
PCT International Publication WO 2007/054931 (published after the priority date of the present patent application), which is assigned to the assignee of the present patent application and whose disclosure is incorporated herein by reference, describes methods and devices for image enhancement in the mosaic domain A mosaic image sensor outputs stream of pixel values belonging to a plurality of input sub-images, each of which is due to light of a different, respective color that is incident on the sensor. An image restoration circuit filters the pixel values in each of the input sub-images so as to generate corresponding output sub-images with enhanced quality, such as with reduced blur. An image signal processor (ISP) then combines the output sub-images so as to generate a color video output image.